Segmenting Chinese in Unicode

The automatic segmentation of Chinese text is an ongoing problem in information retrieval (IR) and computational linguistics: “words” in written Chinese are not delimited by spaces so tokenizing (the first phase of many IR tasks) is considerably more difficult than for Western languages. This paper presents an overview of the segmentation problem, detailing previous research into its solution and introduces Basis Technology’s Chinese Morphological Analyzer (CMA), a new, general purpose hybrid segmentation system. The CMA is Unicode based, and can handle both Simplified and Traditional Chinese text from a variety of locales, including Mainland China, Taiwan, Hong Kong, and Singapore.